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Abstract 

One of the key concepts in testing is that of adequate test sets. A test selection 
criterion decides which test sets are adequate. In this paper, a language schema for 
specifying a large class of test selection criteria is developed; the schema is based on two 
operations for building complex criteria from simple ones. Basic algebraic properties of 
the two operations are derived. 

In the second part of the paper, a simple language — an instance of the general 
schema — is studied in detail, with the goal of generating small adequate test sets 
automatically. It is shown that one version of the problem is intractable, while another 
is solvable by an efficient algorithm. An implementation of the algorithm is described. 
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1 Introduction 



This paper deals with testing of computer programs. However, most of our discussion 
applies to testing of more general systems. 

Testing consists of experiments, called tests, in which the behavior of the system under 
test is compared to its specification. The system is often called an implementation 
under test; the purpose of testing is to conclude whether the system implements the 
specification. 

The test designer must decide, possibly with machine assistance, what tests are to be 
executed and in what order. In this paper we assume that tests are repeatable and that the 
behavior of the implementation under test in each individual test does not depend on the 
order in which the tests are executed. Therefore the test designer's decision is described by a 
set of tests, selected from some set of tests that could be executed. To model this situation, 
we denote by D the test domain, i.e. some given set of tests for the implementation under 
test. Subsets of D are called test sets. 

An important concept is that of adequate test sets. Informally, a subset T of D is 
adequate if we believe that it is sufficient to execute the tests in T, instead of all the 
tests in D. Once we have checked that the behavior of the implementation satisfies the 
specification for each test d in T, we are willing to accept that the same will be true for 
each d in D. To make this concept independent of subjective beliefs, we define adequacy 
with respect to a test selection criterion: A test selection criterion on D is a rule that 
decides for each subset T of D whether T is adequate or not. (Other terms have been used 
in the literature, e.g. data selection criterion ||, test method |J, testing method 0). A 
test selection criterion may be defined based on the knowledge of the implementation under 
test, of its specification, or both; Gourlay || introduced a framework for discussing these 
dependencies explicitly. 

Many natural test selection criteria can be described as follows: There is a collection of 
subsets of the domain D, and T C D is adequate if and only if T intersects every nonempty 
set in the collection. The following three examples of selection criteria from the literature, 
and many others, are of this form. 

1. Condition table method ||. "[I]dentify conditions describing some aspect of the 
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problem or program to be tested" (||], p. 167), and then combine the conditions to 
form test predicates on D, the set of inputs. A test set T is complete (||, p. 170) if 

• for each thus formed test predicate there is a point in T that satisfies the predi- 
cate; and 

• each point in T satisfies at least one of the predicates. 

The first condition is clearly the adequacy of T as described above, with respect to a 
collection of subsets of D. 



2. Cause-effect graphing & 10]. A cause-effect graph is a simplified specification of 



the system under test. Nodes in the graph represent important properties of causes 
(inputs) and effects (outputs) and possibly additional intermediate properties. Edges 
represent how the effects depend on the causes. Once the cause-effect graph has been 
constructed, it can be used for systematic selection of a set of inputs for testing. Let N 
be the set of nodes in the graph. Each input defines a subset of N; thus the domain D 
corresponds to a set of subsets of N. One simple test selection criterion is: 

• Ensure that each effect node is covered at least once. 

This is clearly adequacy as described above, with respect to a collection of subsets 
of D. Myers ( jlCR , pp. 65-68) described a more complex test selection criterion based 
on the cause-effect graph; again his description can be defined as adequacy with 
respect to a collection of subsets of D. 



3. Statement coverage [1C]. Let the implementation under test be implemented by a 



program consisting of a number of statements. For each statement s in the program, 
let X s be the set of the tests in D that cause s to be executed. Then T C D is 
adequate with respect to the collection {X s } if and only if T covers every statement 
covered by D. 

Jeng and Weyuker || give several other examples of test selection criteria of this general 
form, which they call partition testing. 

In the present paper we describe a simple but powerful language for specifying test 
selection criteria; the language is based on our previous proposal [11]. A language for 
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specifying test selection criteria is needed when we wish to free the test designer from dealing 
with individual test cases. The test designer should be able to specify what constitutes an 
adequate test set in a high-level notation, from which individual test cases are then generated 
automatically. 

Balcer, Hasling and Ostrand Q built a system called TSL, which supports this high-level 
approach to testing. Our design can serve as a model for extending the test specification 
language in TSL, and for defining other similar languages. We return to the comparison 
with TSL in Section |7j. 

We describe a general language schema, from which concrete languages are derived by 
choosing types of parameters. The schema is based on two operations for combining selection 
criteria; with these two operations, test selection criteria form a well-behaved algebra. The 
ability to combine criteria using the two operations yields a number of benefits: 

• The language has simple well-defined semantics. 

• The language is powerful — many useful criteria can be expressed in the language. 

• Algorithms that process criteria and generate test sets can use algebraic identities to 
manipulate criteria. 

In the second half of the paper we define one language based on the general schema, 
and study the algorithms that generate adequate test sets for the criteria expressed in 
the language. We show that the problem of finding a minimum adequate test set (i.e. 
an adequate test set of the smallest size) is NP-hard, and then we concentrate on the 
problem of finding a minimal adequate test set (i.e. a test set whose proper subsets are 
not adequate). We also describe what we learned from implementing a prototype tool for 
generating minimal adequate test sets. 

Related work and topics for further research are discussed in the last section. 

2 Example 

To illustrate the concept of a test selection criterion, we now describe a simple testing 
scenario, adopted from the paper by Balcer, Hasling and Ostrand 0. 
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declaration 
separators : { "/", "z" } 
separator _2 : { "/", "x" } 

string_l : { "", "a", "ab", "abed", "abcd987", "abedef ghijklmnopqrstuvwxyz0123" } 
string_2 : { "", "a", "ab", "abed", "abcd987", "abedef ghij klmnopqrstuvwxyzO 123" } 
string_l_occurs : {true, false} 



Figure 1: Parameter declarations for the example 

Test suites typically consist of many test cases that differ only slightly from each other. 
Rather than preparing all the variations one by one, the test designer may prepare a "pa- 
rameterized test case" (a "code template" in the terminology of |2|) and then generate 
individual test cases by systematically filling in the values of the parameters. 

In the sample scenario, a text editor is to be tested against the specification of the 
CHANGE command. The syntax of the command is 
C /stringl/string2 

As in |2), the parameterized test case for this task uses five parameters. (More precisely, 
the TSL description in [Q] uses four parameters and one environment condition; however, 
the distinction is not important for our discussion.) 

Parameter declarations are in Figure [l]. To obtain one individual test case, we select 
one value for each parameter, and substitute the selected values to 

C separator_l string_l separator_2 string_2 
The value of the parameter string_l_occurs is used to set up the current line in the editor 
(so that it does or does not contain string_l). 

Now observe that the parameter declarations in Figure |l| define a test domain D: Each 
combination of values for the five parameters defines a test in D. In some cases it may 
be feasible to execute all tests in D. However, even in our simple example D contains 
2x2x6x6x2 = 288 elements. It is easy to imagine much larger examples, for which 
testing with all inputs in D would be infeasible. The test designer must then select a test 
set, i.e. a subset of D. Sometimes the test designer wants to list the points of the test 
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set explicitly, one by one. However, it is frequently more convenient to write a high-level 
description of a test selection criterion, and let an automated tool select a test set adequate 
for the criterion. 

Let us consider several examples of high-level descriptions of test selection criteria that 
free the test designer from the need to think in terms of individual test cases. For our 
domain D, the criterion 

( string_l = "a" ) (1) 

specifies that the test set must include at least one point in which the value of the parameter 
string_l is "a". The criterion 

EACH( string_l : "a", "ab", "abcd987" ) (2) 

specifies that for each of the three listed values of the parameter string_l the test set must 
include at least one point with that value. It is convenient to have another primitive as an 
abbreviation for EACH whose arguments include all values declared for the parameter; the 
primitive EXHAUSTIVE with one argument has this role. Thus 

EXHAUSTIVE( string_l ) (3) 

has the same meaning as 

EACH( string_l : "", "a", "ab", "abed", "abcd987", "abedef ghijklmnopqrstuvwxyz0123" ) . 

(4) 

As we shall see in the next section, (g) and the criterion 

EXHAUSTIVE( separator^ ) (5) 
can be combined in two basic ways. One combination is 

EXHAUSTIVE( string_l ) <g> EXHAUSTIVE( separator^ ) , 
which can be also written as 

EXHAUSTIVE( string_l, separator.! ) . 
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It specifies that all possible combinations of the values of string_l and separator _1 must 
be included; since string_l assumes six values and separator_l two values, any test set 
adequate for this criterion must contain at least 12 elements. The other combination of (|3|) 
and is 

EXHAUSTIVE( string_l ) tti EXHAUSTIVE( separator^ ) , 

which merely requires that the test set must be adequate for (|3|) and also for (||). A test 
set containing 6 points is sufficient for that; for example, the following six combinations of 
string_l and separator_l are sufficient: 

string_l separator_l 

2. "a" "/" 

3. "ab" "/" 

4. "abed" "/" 

5. "abcd987" "/" 

6. "abcdefghijklmnopqrstuvwxyz0123" "z" 



In the next section we describe a more systematic approach to the construction of test se- 
lection criteria. We shall see that many complex criteria, including EACH and EXHAUSTIVE, 
may be constructed from simple ones. 



3 A language for test selection criteria 

3.1 A general language schema 

We are now going to describe a language for specifying instances of the test selection prob- 
lem. We start by describing a general language schema. Many different concrete lan- 
guages may then be obtained from the schema by allowing different parameter types. One 
such choice of parameter types and the resulting concrete language are discussed in Sec- 
tion |3.3| and in the rest of the paper. 

To define an instance of the test selection problem, we have to specify a domain D and a 
test selection criterion on D. In our approach, D and the criterion on D have the following 
special form: 
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N 



• D is a subset of the Cartesian product P = TT Qi of certain sets Q±, . . . , Qn- The 

i=l 

points in P are vectors (v%, . . . , vn) of parameter values Vi £ Qi. 

• The criterion is defined by a set of subsets of P. 

Thus to define an instance of the test selection problem, we specify sets Qi , a subset 
D of the product P of Qi , and a set of subsets of P. In our language, the specification 
consists of three parts: 

1. declaration of parameters; 

2. a constraint; 

3. a test selection criterion. 

Part 1 defines the sets Qi , part 2 the set D, and part 3 the set of subsets of P. 
The first part, denoted A, is a set of declarations 

Qi ■ Qi 

each of which declares a parameter qi and its range Qi . Define 

N 

P(A) = HQ l . 

1=1 

For example, for the declarations in Figure B, -P(A) is the Cartesian product of five 
sets Qf. 

Q l = {"/","z"} 
Q 2 = {"/","x"} 

Q 3 = Q 4 = {"","a","ab","abcd","abcd987","abcdefghijklmnopqrstuvwxyz0123"} 
Q5 = {true, false} 

The second part is a constraint; it is a boolean expression ip = ip(qi, ■ ■ ■ , qat) built from 
primitive constraints by means of binary operators V (logical or) and A (logical and). To 
interpret the constraint, we have to assign the value true or false to each primitive constraint 
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in the expression when arbitrary values (vi, . . . , ujv) are substituted for the parameters 
(qi, . . . , gjv). The constraint then defines the domain 

D(A, np) = {(vx,...,v N )e P(A) | • • • , ujv) = true } . 

We write D(ip) instead of D(A,ip) when no misunderstanding is possible. 

The example in Section ^ does not specify any constraint, and therefore D(ip) = -P(A). 

The third part is a test selection criterion; it is an expression built from primitive 
criteria by means of binary operators tt) and ®. The value of such an expression T is a 
set S(A,r) of subsets of -P(A). Again we write S(T) instead of S(A,T) when no misunder- 
standing is possible. Once the value S(T) has been defined for every primitive criterion T, 
we define S(T) for general T as follows: Given two criteria T± and r 2 , define 

s(riwr 2 ) = {x 1 xeS(ri) or xeS(r 2 )} = s(ra)us(r 2 ) , 
S(r!®r 2 ) = {iini 2 | ^eS^eS^)} . 

In our example in Section ^, when T is the primitive criterion 

( string_l = "a" ) 

the set S(r) contains a single subset of -P(A), namely 

{ (sep 1 ,sep 2 ,si,s 2 ,o) G P(A) | s x = "a" } . 

Similarly, we could take EACH and EXHAUSTIVE as primitive criteria and define their 
values S(r); however, we shall see later that these criteria can be derived from simpler ones 
using tt) and <g). 

Definition. An instance of the test selection problem is I = ( A, ip, T ), where 
A is a set of parameter declarations, ip is a constraint, and T is a test selection criterion. 
A set T C D(A,ip) is adequate for / if T n X ^ for every X € S(A,T) such that 
X n D (A, ip) ^ 0. We also say that T is adequate for T if A and if; are understood from 
the context. 

From the definition of TittH^ it follows that a test set T is adequate for rittir 2 if and 
only if it is adequate for T\ and also for T 2 . The criterion TittT^ is used when the test 
designer wants to satisfy T\ and T 2 independently. 
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The criterion I?!®^ is used when the test designer suspects dependencies between I\ 
and T2, and wants to test for the faults produced by combinations of causes. If T\ enforces 
the selection of a test point that has some property p\ and T2 the selection of a test point 
that has some property P2, then the criterion TitS^ enforces the selection of a test point 
with the property pi and P2 (if such a point exists in D(ip)). 

Since S(r) is the value of the expression T, it is natural to write Ti = T2 when S(Ti) = 
S(T2), and T\ C T2 when S(ri) C S(r2). It is a simple exercise to show that both t+J and (g) 
are commutative and associative, and that the following distributive law holds: 

(ri u r 2 ) ® r 3 = (r l ® r 3 ) w (r 2 ® r 3 ) . 

Since l±l and <g> are associative, we write expressions like ritfcll^tfcira and ritgir^fSTs 

m 

without parentheses. We also use the notation l+J Tj for rilill^W . . . ttir m , and similarly 
for (g). 

3.2 Comparing criteria 

In this section we define several relations for comparing test selection criteria. The defini- 
tions of this section are not used in the rest of the paper, but the concepts will illustrate 
some important properties of the algebra of test selection criteria. 

The following relation C describes the notion that one criterion is less stringent than 
another. 

Definition. Let Si and S2 be two sets of subsets of a set P. Write Si C S2 if the 
following is true for every TCP: if T n X 7^ for every nonempty X E S2 then T Pi X 7^ 
for every nonempty X € Si. Write Si ~ S2 if Si C S2 and S2 Q Si. For a fixed A and criteria 
ri and T 2 , write Ti C T 2 if S(A, r x ) C S(A, T 2 ), and Ti ~ T 2 if S(A, r x ) ~ S(A, T 2 ). 

The proof of the following proposition follows directly from definitions. In view of part 1, 
Ti E 1^2 if and only if (A, true, 1^2) subsumes (A,true,ri) in the terminology of Hamlet QJ. 

Proposition 3.1 Let A be a fixed set of declarations. IfT\ and T2 are two criteria then 
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1. T\ C r 2 if and only if every T C P(A) adequate for (A, true, r 2 ) is a/so adequate for 
(A,true,ri); 

2. Ti ~ r 2 if and only if (A, true, Ti) and (A,true,r2) have the same adequate sets; 

3. Fx C T 2 implies T 1 QF 2 . □ 

By part 3, T\ = T 2 implies ri~T 2 . Although ri ~ T 2 does not imply T\ = T 2 , 
Proposition below shows that ~ and = are closely related. 

Let S be a set of subsets of a set P. A set X E S is minimal in S if X ^ and 

Y e S, Y C X implies Y = X or Y = . 

Let MIN(S) be the set of all minimal IeS. 

Proposition 3.2 If S, Si and S 2 are finite sets of subsets of P then 

1. MIN(S)~S ; 

2. Si ~ S 2 if and only if MIN(Si) = MIN(S 2 ) • 

Proof. 1. Since MIN(S) C S, it follows that MIN(S) CS. Since S is finite, for every 
nonempty Y € S there exists a minimal X € S such that X C Y; therefore S C MIN(S) by 
the definition of C . 

2. If MIN(Si) = MIN(S 2 ) then by part 1 we get 

Si ~ MIN(Si) = MIN(S 2 ) ~S 2 . 

Assume Si ~ S 2 and Xi G MIN(Si). We have MIN(Si) ~ MIN(S 2 ) by part 1. Set Ti = P\Xd 
thus T 1 HX 1 = 0. By the definition of MIN(Si) C MIN(S 2 ) there exists X 2 G MIN(S 2 ) such 
that T\ n X 2 = 0. Thus X 2 C Xi. Now by the same argument applied to T 2 = P \ X 2 
there exists X[ G MIN(Si) such that I[ C I 2 C 1^ Since Xl is minimal in MIN(Si), we 
have Xi = X{, and therefore Xi = X 2 . We have proved that every X\ G MIN(Si) belongs 
to MIN(S 2 ). By symmetry we get MIN(Si) = MIN(S 2 ). □ 
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It is easy to verify that 

Ti C ri implies r x W T 2 Q T[ tt) T 2 
Ti ~ ^ implies Ti W T 2 ~ ri tt) T 2 

However, Ti ~ ri does noi imply Ti®^ — T'^Y^- Thus, even for ip = true, to deter- 
mine which test sets are adequate with respect to (A, ip, T\<S>T2), it is not enough to know 
which test sets are adequate with respect to (A, tp, Fi) and which are adequate with respect 

to (A,v,r 2 ). 

3.3 Enumerated types 



From the general language schema described in Section 3.1 we obtain a concrete language 



by specifying allowed parameter types. To specify a parameter type, we must describe 

• the range; 

• primitive constraints; 

• primitive criteria. 

In addition, we must supply rules to evaluate primitive constraints and primitive criteria, 
so that D(tf)) and S(T) are defined for any ip and V. 

We use the following convention: If <p = ip(qi, . . . , qw) is a Boolean expression then (<p) 
is the criterion for which the value S((y)) contains a single subset of P(A), namely 

{ (vi, . . . ,ujv) G -P(A) | (p(v 1} . . . ,v N ) } . 

In the rest of the paper we work with one concrete language obtained as follows: Each 
parameter range is a finite set, which is explicitly listed in the declaration. Each primitive 
constraint has one of the two forms 

q% = Cj 
qi / ^ 
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where qi is one of the declared parameters, and Cj is one of the values in the range of it 
is obvious how these constraints evaluate to true or false. Each primitive criterion has one 
of the three forms 

{qi = ci) 
ANY.TEST 

where qi is one of the declared parameters and Q is one of the values in the range of q^. 
The values S((qi = Cj)) and S((qi 7^ q)) are defined by the convention at the beginning of 
the previous paragraph. The value S(ANY_TEST) contains only the set -P(A) itself. 

The present definition of S(T) differs slightly from the definition of the "pile assigned 
to r" in the previous design of the language |fi"l] ]; namely, we do not require that € S(T) 
and -P(T) E S(T). We find the present definition technically more convenient. 

Using these primitive criteria and the tt) and (g> operations, the test designer can write 
down many other useful criteria. In particular, it is possible to specify that a particular 
vector ... ,vn) of parameter values Vi € Qi must be included in the selected test set. 
For example, to ensure that the vector in which 

separator_l = separator_2 
string_l 
string_2 
string_l_occurs 

is in the selected set, the test designer would use the criterion 

( separator_l = "/" } <8> ( separator _2 = "/" ) &> ( string_l = "abed" ) ® 
( string_2 = "ab" } ® ( string_l_occurs = true } . 

The criteria EACH and EXHAUSTIVE, which were informally described in the previous 
section, can also be constructed using l±) and (g). The general definition is as follows: Let Qi 



= "abcd : 
= "ab" 
= true 
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be the range of the parameter If Y C Qj then define 

EACH( % : Y ) = |±) < ft = a) . 

The criterion specifies that each value in Y must be tested (as long as there is at least one 
point in D(ip) with that value of qi). 

For any sequence q^, qi 2 , . . . , qi m of parameters, define 

m 

EXHAUSTIVE^ , . . . ,q im ) = ® EACH( % : Q<. ) . 

This specifies that all the combinations of values of q^ , qi % , . . . , qi m allowed by the constraint 
must be tested. 

The criteria EXHAUSTIVE^, . . . , qjy) and ANY_TEST are at opposite ends of the scale 
ordered by C. Only the set D(ip) itself is adequate for EXHAUSTIVE(qi, . . . , q^). If 
D(ip) ^ 0, any nonempty subset of D(ip) is adequate for ANY_TEST. 

4 Worst-case complexity of two test selection problems 

In this section we work with the concrete language from Section and we consider 
algorithmic aspects of the criteria specified in the language: Given one such criterion, how 
difficult is it to find an adequate test set that is in some sense "small"? 

4.1 Two basic problems 

Let I = (A,ip,T) be an instance of the test selection problem, and let T be an adequate 
test set for I. Say that T is a minimum adequate test set if no set of cardinality smaller 
than \T\ is adequate. Say that T is a minimal adequate test set if no proper subset of 
T is adequate. 

We are interested in algorithms for two problems: 

The Minimum Adequate Set Search Problem (Mum AS) 

Input: An instance I = (A,ip,T). 
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Output: A minimum adequate test set for I. 

The Minimal Adequate Set Search Problem (Ma IAS) 
Input: An instance I = (A,ip,T). 
Output: A minimal adequate test set for /. 

The size of the instance / = (A,ip,T), denoted |/|, is the total length of the dec- 
larations in A and of the expressions ip and V. Often the cardinality of the set -P(A) is 
exponential in the number of parameters in A. For example, if each parameter range Qi , 
1 < i < N, consists of two values then the cardinality of -P(A) is 2^. Thus the cardinality 
of a minimum or minimal adequate test set T may be exponential in in that case no 
algorithm that outputs T can execute in time polynomial in |/|. We shall therefore measure 
the execution time of such algorithms in terms of \I\ + |T|. Thus a polynomial-time algo- 
rithm for Mum AS or Ma IAS is an algorithm whose worst-case execution time is bounded by 
a polynomial function of \I\ + \T\. 

We shall identify two obstacles on the path toward efficient algorithms for MumAS and 
Mai AS. One obstacle, related to the boolean satisfiability problem, applies to both MumAS 
and MalAS (section |4.2| ); the other, related to graph colorability, applies only to MumAS 
(section |Q| ) . 

4.2 Connections with boolean satisfiability 

For classifying problems as NP-complete, NP-hard, etc., we use the terminology of Garey 
and Johnson 0]. MumAS and MalAS are search problems ([||, p. 110). The following 
decision problem will be useful in our analysis of the complexity of MumAS and MalAS. 

The Empty Adequate Set Problem (EA) 
Input: An instance I. 

Question: Is the empty set adequate for I? 

Denote by (J S(r) the union of all sets in S(r). The empty set is adequate for / = (A, ijj, T) 
if and only if D(ip) n U S(T) = 0. 

It is not difficult to prove that EA is in co-NP. However, we are more interested in 
proving that EA is NP-hard; we now prove the NP-hardness of EA, by reduction from the 
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boolean satisfiability problem. 

Theorem 4.1 The problem EA is NP-hard, even if the input I = (A,ip,T) is such that 

1. ifi = true, or 

2. r = ANY.TEST. 

Proof. By reduction from 3SAT (0], p. 46). Let C be an instance of 3SAT. We construct 
an instance I such that C is satisfiable if and only if is not adequate for /. 

Let C = {ci, C2, • • • , c m } be a set of clauses on a finite set C/ of boolean variables, such 
that 

Cj = dji V Clj2 V Oj3 

for 1 < j < m. Each literal ajfc is either a variable u in [/ or its negation u. Let A be the 
declarations 

u : { true, false } 

for u in U. 

For ip = true, the empty set is adequate if and only if {J S(T) = 0. Define 

m 

r = ® (r^ar^ar^) 

where 

{(u = true) if Ojfc = ii 
(u = false) if = u 

for k = 1,2, 3, and define / = (A, true, T). Then C is satisfiable if and only if |J S(r) ^ 0. 
For T = ANY_TEST, the empty set is adequate if and only if D(tp) = 0. Define 

m 

*l> = A (^il V ^"2 V ^3) 
5=1 
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where 



u = true if cu^ = u 



u = false if ajk = u 

and define I = (A, ip, ANY.TEST). Then C is satisfiable if and only if D(ip) ^ 0. □ 



The following lemma shows that any lower bound for the execution time complexity of 
EA implies a lower bound for Mum AS and Ma IAS. 

Lemma 4.2 Let w be an integer function of an integer variable such that the value w(i) 
for any integer i can be computed in 0(w(i)) steps. If there exists an algorithm for MumAS 
or MalAS that for every input I produces an output T in at most w(\I\ + \T\) steps, then 
there exists an algorithm that solves EA for every input I in 0(w(\I\)) steps. 

Proof. To solve EA on input I, compute and execute the algorithm for MumAS (or 

MalAS) on input / for at most steps. The answer to the question in EA is "yes" if the 

algorithm terminates with output T = 0. The answer is "no" if the algorithm terminates 
with output T ^ or does not terminate in steps. □ 



Theorem 4.3 If P ^ NP then neither MumAS nor MalAS is solvable by a polynomial-time 



algorithm, even in cases 1 and 2 in Theorem 4-1 



Proof. Apply Theorem 4.1 and Lemma 4.2. 



□ 



4.3 Connections with graph colorability 

We have identified one reason why MumAS and MalAS are difficult: tp and T may encode 
arbitrary boolean expressions, and thus any algorithm for MumAS or MalAS can be used to 
construct an algorithm for 3SAT. It is therefore natural to ask whether MumAS and MalAS 
become easier when ip and V belong to a smaller class of expressions. 



18 



We start with a simple such class, the criteria in l+J(g)= form. The W<8>= form of a test 
selection criterion is 

l±J <g> i> (6) 

3=1 k=l 

where Tjj~ are primitive criteria of the form (qt = Cj). Define an instance I = (A,ijj,T) 
to be simple if tp = true and T is in ttl(g>= form. In the next section we shall see that 
the problem MalAS for simple instances is solvable by a polynomial-time algorithm. In 



contrast, MumAS for simple instances is NP-hard, as will be established in Theorem 4.8. 
The following decision problem will be used in the proof. 

The Minimum Adequate Set Problem for Simple Instances (MASI) 
Input: A simple instance / and an integer K. 
Question: Is there a set T adequate for / such that |T| < K ? 

We are going to show that MASI is equivalent to GRAPH K-COLORABILITY (§], p. 191). 

Let S be a set of sets. The intersection graph of S is the graph G = (S, E) in which 
the set of vertices is S and the set of edges is 

E = { {X, Y} | X,Y eS, X and X n Y + } . 

When G = (V, E) is a graph, the complement of G is the graph G = (V, E), where 

E = { {x, y} | x, y G V, x ^ y and {x, y} E } . 

The proof of the following simple lemma is left to the reader. Note that the lemma 
would not be true if we admitted primitive criteria of the form (g, ^ Cj). 

Lemma 4.4 Let I = (A, true, T) be a simple instance, and let So Q S(T). If X C\Y ^ for 
all X, Y G S then f|{A|A G S } ^ 0. □ 

By the lemma, a set So C S(T) forms a clique in the intersection graph of S(T) if and 
only ifn{A|AGS o }y^0. 
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Proposition 4.5 Let I = (A, true, T) be a simple instance, let G be the complement of the 
intersection graph o/S(T) \{0}, and let K be an integer. The graph G is K -colorable if and 
only if there exists a set T adequate for I such that \T\ < K. 

Proof. Assume G = (S(r) \ {0},i?) is i^T-colorable. This means that there exists a 
mapping / : S(r) \ {0} -»■ {1, 2, . . . , K } such that f(X) / f(Y) when {X, Y} G E. Define 

S i = {ieS(r)\{f)}|/(l)=i} 

for j = 1, 2, . . . , K. From the definition of the intersection graph we get that if X, Y £ Sj 
then X n Y ^ 0; by Lemma we have p|{^l^ G Sj} ^ 0. Form a set T by choosing one 
point in each £ Sj}, j = 1, 2, . . . , K . Thus \T\ < K and T intersects each nonempty 

X € S(r), which means that T is adequate for /. 

Conversely, assume that there exists a set T adequate for / such that \T\ < K. Write 
T = {d u d 2 , ■ ■ ■ , d K } and define a K-coloring / : S(r) \ {0} -> {1, 2, . . . ,K} of G by 

f(X) = min{ j | dj e X } . 

Since T is adequate, / is defined for each X € S(r) \ {0}. If f(X) = f(Y) = j then 
dj € X fl Y, hence X n Y ^ 0, hence {X, is not an edge in G. Thus / is a coloring of G. 

□ 

Proposition 4.6 For eac/i graph G = (V, E) there exists a simple instance I = (A, true, V) 
such that the intersection graph of S(T) is (isomorphic to) G. The declarations A consist 
of one boolean parameter for each vertex in V. 

Proof. Let V consist of iV vertices, V = {xi, X2, . . . , xjv}- Let A be the declarations 

qi : {true, false} 

for i = l,2,...,N. Define 

H(i) = {j e{l,2,...,N} | j^i and {x uXj }^E} , 
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N 

r = l+l ( (<?i = tru e) ® <g> = false) ) . 

i=i jetf(i) 

Then S(r) consists of the sets 

Xi = { (vi, . . . , Ujv) £ -P(A) I u i = true an d G : = false } 

for « = 1,2, ... ,N. We have Xj 7^ Xj for j 7^ j, the sets Xj are nonempty, and the 
mapping X, 1— > Xj is an isomorphism between G and the intersection graph of S(T). 

□ 



By Propositions and H GRAPH K-COLORABILITY and MASI are polynomially 
equivalent. From known results for GRAPH K-COLORABILITY (§], p. 191) we obtain the 
following result for MASI. 



Theorem 4.7 The problem MASI is NP-complete, even for K = 3. 



□ 



It remains to transform MASI into MumAS. The only potential complication is that 
"polynomial" means "polynomial in the size of input" for MASI and "polynomial in the 
size of input and output" for MumAS. However, if the input instance I is simple and the 
output set T is minimum then \T\ is bounded by Indeed, for criterion @ there exists 
an adequate test set of cardinality at most m, which means that the cardinality of the 



minimum set T is also bounded by m. Thus Theorem [4.7| yields the following result for 
MumAS. 

Theorem 4.8 If P ^ NP then MumAS is not solvable by a polynomial-time algorithm, 
even for simple instances. □ 



In view of Theorem 4.5 , we are not likely to find a polynomial-time algorithm for M umAS. 
It is still possible that there is an algorithm for MumAS that is efficient in some other sense, 
but we have not been able to find any such algorithm. However, in the next section we 
present a practical algorithm for Mai AS. 
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The transformation in Propositions 4.5 and |4.6| yields more than results for MASI and 
Mum AS. For example, if we had an algorithm that for every simple instance would find 
an adequate test set whose cardinality is within the factor (1 + e) of the minimum, then 
we would also have an algorithm to color any graph with the number of colors within the 
factor (1 + e) of the minimum. No such polynomial-time algorithm is presently known for 
any fixed constant e. 



By virtue of Proposition 4.5, any algorithm for graph coloring can be transformed into an 
algorithm for constructing adequate test sets for simple instances; when the graph coloring 
uses the minimum number of colors, the adequate test set is minimum. Many heuristic 



algorithms for graph coloring have been studied; see e.g. [12, [bj and the references therein 



However, we are interested in the problems MumAS and MalAS, rather than MASI; the 



restriction to simple instances is severe. We have already noted that Lemma 4.4 does not 



hold if primitive criteria [qi ^ Cj) are allowed. Moreover, if ip is a general constraint then 



Lemma L4 may fail for the sets in S(r) restricted to the domain D(ip). 



5 Algorithms for finding minimal adequate test sets 
5.1 An algorithm for normalized instances 



In this section we concentrate on the problem MalAS defined in Section 4J.. We start with 
an efficient algorithm for the input instances I = (A, ip, T) in which ip and V belong to a 
certain restricted class of expressions. Afterwards we show how to use the algorithm for 
general instances. 

The test selection criterion 

l±J <g> I> , (7) 

j=l k=l 

where T-^ are primitive criteria, is said to be in the ttJ® form. The constraint 



V A *jk > ( 8 ) 

i=i k=i 
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where ipj^ are primitive constraints, is said to be in the VA form. (This is also called the 
disjunctive normal form.) 

An instance I = (A, ip, T) is normalized if ip is in the VA form and T in the til® form. 

Let A be a fixed set of parameter declarations qi : Qi, i = 1, 2, . . . , N. We say that a 
set X C -P(A) is a subcube if it is in the form Y\.f=i Ri where Ri C Qi. For our concrete 



language of Section 3.3, every X G S(T) is a subcube. When the criterion V is in the tfcl 
form, it is easy to compute the set S(T): The subcubes in S(r) correspond to the terms 
®]Sjk m ©• Similarly, every term Ak^jk i n ® defines the subcube D(/\ k ipj k ), and 

0(ViA* ^i*) = Uj D(Ak 1> jk ). 

The algorithm in Figure |2| constructs a minimal set adequate for a given normalized 
instance. The input for the algorithm consists of two sets of subcubes: the set S = S(r), 
and the set 

C = {D(A k fj, jk ) \j = 1,2,..., m} 

for the constraint (^). When the algorithm terminates, the set variable T contains a minimal 
adequate set. 

In the program for the algorithm, forall denotes iteration over all elements of a set in 
some arbitrary order. The values of the data type "point" are the elements of -P(A). The 
function call Find_point(X, C) finds a point in the set X n{JC; if the set is empty, the 
function returns NIL. 

For each t G T, the variable contains(t) stores a set of subcubes; a subcube X G S 
belongs to contains{t) if and only if t G X. For each X G S, the variable count(X) stores 
the cardinality of X D T. 

The algorithm works in two phases: The first phase finds an adequate test set, and the 
second phase trims the set to make it minimal. 

When sets are represented as arrays or linked lists, adding one element takes constant 
time, and iterating through a forall loop adds only constant time per iteration. The deletion 
operation on the last line of the program is implemented by marking the element as deleted; 
that also takes only constant time. 

When points and subcubes are represented as sorted lists of primitive constraints, the 
function Find_point(X, C) and the test "if t G Y" are implemented by a single pass through 
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inputs 

S : set of subcube 
C : set of subcube 



variables 

T : set of point 

contains(t) : set of subcube, for t G T 

count(X) : integer, for X G S 

initially 

T = 

count(X) =0, for X £ S 

program 

forall X G S do 

if count(X) = then 
t := Find_point( X, C ) 
if t ^ NIL then 
T :=TU{t} 
forall y € S do 
if i G Y then 

contains(t) := contains(t) U 
count(y) := count(Y) + 1 

forall t G T do 

if max( count(Y), Y £contains{t) ) > 2 then 
forall Y G contains(t) do 
couni(Y) := countiY) — 1 
T:=T\{£} 



Figure 2: Algorithm for MalAS 

the lists representing the two arguments. Adding it all up, we get the bound 0(|I| 2 ) for the 
total execution time of the algorithm on any input instance /. We summarize our analysis 
in a theorem. 

Theorem 5.1 There is an algorithm to solve the problem MalAS for any normalized in- 
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stance I in time 0(\I\ 2 ). 



□ 



5.2 The cost of normalization 

Requiring input instances to be normalized would be inconvenient to the users. For example: 

• The constraint is often naturally specified in the conjunctive, rather than disjunctive, 
normal form. 

• The user should be able to take any two criteria and combine them by means of (g>. 
The resulting criterion is not in the W® form. 

Therefore our design allows users to specify the instance in the general form defined in 
Section ||. The instance is automatically converted into an equivalent normalized form 
before the algorithm in Figure [2] is applied. 

The normalization is easy to implement. The well-known procedure transforms boolean 
expressions into VA form by repeatedly replacing a conjuction of disjunctions by an equiv- 
alent disjunction of conjunctions. By virtue of the distributive law for ttJ and ®, the same 
procedure works for the test selection criteria built using ttl and ®. 

However, the user should understand that the normalization may in some cases be 
expensive, in terms of execution time. In the worst case, the execution time is exponential in 
the size of the original expression. We shall now discuss the implications of the normalization 
cost, separately for the constraint expression ip and for the criterion expression T. 

For ip, the exponential increase of the execution time of the normalization procedure is 
more common and more serious than for T. A large instance / = (A, ip, T) of the test se- 
lection problem is typically obtained by putting together several instances Ij = (Aj,ipj,Tj) 
with disjoint sets of parameters. It is then natural to take ip = AjV'j- If m independent 
constraints are put together to form 

in 

i> = A C^-i v 

then the equivalent VA form of t/j has 2 m terms. Thus in this case the total execution time 
is at least proportional to 2 m , even if the test set produced at the end is very small. 
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The cost of normalizing the criterion T is less critical; in most cases large criteria lead to 
large test sets. However, "in most cases" does not mean "always", as the following example 
shows: 

Example. Let A consist of N declarations qi : {0, 1}, 1 < i < N. Consider the criterion 



N N 

<g> l±J (ft = 0) . (9) 

J=l i=l 

The equivalent l±l<8> form is 

l±J <g> <® = o> , (io) 

A ieA 

where A runs through all nonempty subsets of {1,2, .. . , N}. The only minimal adequate 
test set is T = {(0,0,... ,0)}, of cardinality 1. In transforming (||) to ([To|) the algorithm 
generates all the 2^ — 1 expressions 



(ft = 0) , 

ieA 



where ^ A C {1,2,... ,N}. □ 



Nevertheless, we conjecture that, in most practical situations, if the test designer specifies 
a selection criterion whose equivalent WCED form is very large, then every adequate test set 
will also be very large. In such cases long execution time (at least proportional to the size 
of the produced test set) cannot be avoided. 

In the next section we shall show that for the instance / built by combining independent 
instances Ij, we can solve the problem Ma IAS separately for each Ij and then put the 
solutions together to produce a test set adequate for /. We shall also describe an algorithm 
for decomposing instances into independent components. We expect that for most instances 
of the test selection problem arising in practice the decomposition method will avoid the 
exponential cost of normalization. 
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5.3 Decomposition of instances 

When the test designer constructs a large instance of the test selection problem, it is likely 
that the instance is built from subproblems that are in some sense independent. Now we 
show how such structure can be exploited to construct minimal adequate test sets. 

Definition. Two instances Ij = (Aj,ipj,Fj), j = 1, 2, are independent if no parameter 
occurs in both Ai and A2. 

Definition. Let Ij = (Aj,ipj,Fj), j = 1,2, be two independent instances. Let A = 
Ai U A2. Define two instances 



When a is A <8> or A l±J, we say that I\ and I2 form an independent a- decomposition (or simply 
a decomposition) of ii[a]i2- 

We now construct adequate test sets for I\[f\®\l2 and 2i[AW]i2 from adequate test sets 
for Ii and I2. For two nonempty sets T\ and T2 such that |Ti| = m, |2~2 1 = n, define the set 
T\ || T 2 C Ti x T 2 as follows: Let T x = {n, r 2 , . . . , r m }, T 2 = {s 1: s 2 , . . . , s n }, and 



Thus the definition of T\ || T2 depends on the order in which we number the elements of T\ 
and T2; we assume that one such order is chosen arbitrarily. 



Definition. Let Ij = (Aj,ipj,Fj), j = 1,2, be two independent instances such that 
D(ipj) ^ 0, and let Tj C D(ipj) for j = 1,2. Define 



/i[A®]J 2 
/i[AW]/ 2 



(A,^i AV 2 ,ri®r 2 ) 
(A,Vi AV 2 ,riwr 2 ) 




s„) } if m > n 
n ,s n ) } if m < n 



if m = n 



Ti[A®]T 2 = T 1 xT 2 

'0 if Ti = = T 2 



{si}xT 2 ifT 1 = 0/T 2 
, Ti || T 2 if T x / / T 2 

where Sj G D(tpj), j = 1,2, are some arbitrarily chosen elements. 
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Theorem 5.2 Let Ij = (Aj,ifjj,Tj), j = 1,2, be two independent instances such that 
D(ipj) 7^ 0, and let a be A® or AW. If Tj is an adequate test set for Ij, j = 1,2, then 
T\[a\T2 is an adequate test set for ii[a]/ 2 . If Tj is a minimal adequate test set for Ij, 
j = 1,2, then Ti\a]T2 is a minimal adequate test set for Ii[a]l2- 

Proof. Let A = AiUA 2 , Pi = P(Ai), P 2 = P(A 2 ), P = P(A), and V = ^iAV> 2 - Thus 

P = Pi x P 2 
£(A,y>) = D(Ai,Vi)xI>(A 2 ,^ 2 ) 
S(A,Ti) = {XixP 2 |Xi€S(Ai,ri)} 

S(A, r 2 ) = { ?i x i 2 1 x 2 e S(A 2 , r 2 ) } 
s(A,riwr 2 ) = s(A,ro u s(A,r 2 ) 
s(A,ri^r 2 ) = {x 1 xx 2 |x 1 eS(A 1 ,r 1 ), x 2 eS(A 2 ,r 2 )} 

Let Tj be an adequate test set for Ij, j = 1, 2; that is, PjHX / whenever X G S(Aj,Fj) 
and X n D(Aj,%bj) + 0. Let T = Ti[a]T 2 . 

For a = A®, if X x xX 2 G S(A,Fi®r 2 ) and (X x xX 2 )nP>(A, V) / then Xj G S(A j ,r j ), 
Xj n D(Aj, %bj) + 0. Therefore Xj r\Tj^% and (X x x X 2 ) n T ^ 0. 

For q = Atbl, if Xi x P 2 G S(A,Ti) and (X 1 x P 2 ) nP>(A,V) + then X x G S(Ai,ri), 
Xi n D(Ai,V'i) + 0- Therefore Xi n Ti ^ and (Xi x P 2 ) n T ^ 0. The argument for 
Pi x X 2 G S(A,T 2 ) is symmetrical. 

Now let Tj be a minimal adequate test set for Ij, j = 1, 2, and let T = Ti [a]T 2 . 

Let a = A(g>. To prove that T is minimal, take any (ii,i 2 ) G T. Since Tj is minimal, 
there is Xj G S(Aj,i/jj) such that Xj n D{Aj,^jj) ^ and Xj n (Tj \ {^}) = 0. For 
X = Xi xX 2 we have InD(A^) ^ and Xn(T\ {(ti,i 2 )}) = 0. Therefore T\{(ti,t 2 )} 
is not adequate. Thus T is minimal. 

Let a = AW. If Ti = T 2 = then T = 0, hence T is minimal. Now assume, without loss 
of generality, that |Ti| > |T 2 | and Ti 7^ 0. Then for every t\ G T\ there exists exactly one 
t 2 G D(ip2) suc h that (ti,t 2 ) G T. To prove that T is minimal, take any (ii, t 2 ) G T. Since Ti 
is minimal, there is X l G S(Ai, Y>i) such that X x n D(Ai, V'i) + and X x n (Ti \ {t{\) = 0. 
For X = Xi x P 2 we have X n D{A,^) ^ and X n (T \ {(ti, * 2 )}) = 0. Therefore 
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T \ {{t\i £2)} is not adequate. Thus T is minimal. 



□ 



One can prove that if T\ and T2 are minimum adequate then Xi[ALfcl]T2 is also minimum. 
However, the same is not true for Ti[A®]l2, as the following example shows: 

Example. Define two instances Ij = (Aj,ipj,Tj), j = 1,2: The declaration Aj is 

Xj : {1,2,3} , 

there is no constraint (i.e. ipj = true), and the criterion Tj is 

(xj / 1) tfc) ( Xj ^ 2) t±J (xj / 3) . 

If T\ and T2 are minimum adequate sets for I\ and I2 then |Ti| = | T2 1 = 2, hence \T\ x 
T2I = 4. However, the three-element set {(1, 1), (2, 2), (3, 3)} is adequate for 7i[A<8>]/2- 

□ 



To utilize Theorem :l2 in constructing minimal test sets, we simply add the operations 
[A<8>] and [Atfcl] on instances to the language. The test designer may then specify a large 
instance as a combination of smaller components, using [A(g>] and [Atfcl]. In fact, if the 
language has appropriate scoping rules for the names of parameters then we need not 
require that the parameter names in the component instances be different. 

Now we describe a simple algorithm for discovering a decomposition into independent 
instances, when the decomposition is not explicitly specified by the test designer. The 
algorithm works on the instances / = (A,ip,T) in which ijj has the form /\ k ip k . The 
algorithm groups some terms ip k and some subexpressions of V together, but does not 
attempt to use distributive laws to transform the expressions ip and V. 

Consider an instance / = {A,ip, V) in which A consists of declarations qi : Qi, i = 
1, 2, . . . , N. Subexpressions (often called well- formed subexpressions) of T correspond 
to subtrees of the parse tree of T. For i = 1, 2, . . . , N, let T(i) be the smallest subexpression 
of r that contains all occurrences of qi in T; in the parse tree of T, T(i) corresponds to the 
smallest subtree containing all the leaves labeled (qi = c) and (qi ^ c). 

Define two binary relations W^l and Wy on the set {1, 2, . . . , N}: 
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• i Wjj j if i and j occur in ip k , for some k; 

• i W"r i if i occurs in T(i). 

Let W be the finest equivalence relation on {1,2,..., N} such that W 5 U Wp. Com- 
puting W is a straightforward application of the transitive-closure algorithm (O, p. 199). 
Now each equivalence class B of W determines a subset of the declarations A; the 
subsets Ab are pairwise disjoint. By the construction of W we have 

^ = A A V'fc 

B keB 

where A i s the conjunction over the equivalence classes B of W . Each equivalence class 

B 

B determines a subexpression Tb- The expression T is formed from by means of ttl and 
(8>. Thus we have decomposed I into independent instances Ib, from which / is formed by 
means of [Attl] and [A®]. 

It is of course possible that iW j for all i,j S {1,2,... , N}. In that case this simple 
approach to decomposition does not help. However, in those cases where I has been formed 
by combining several independent instances using [Attl] and [A®], the algorithm will lead 
back at least to the original independent instances, and it may even discover a decomposition 
into smaller instances. 

5.4 Generalized decomposition 

In analogy to the operations [A®] and [Attl], we can also define 

ii[V<g>]I 2 = (Ai U A 2 , ipi V if} 2 , TiigT^) 

Ji[vw]j 2 = (Ai u a 2 ,Vi v^ 2 ,ri«r 2 ) 

whenever Ij = (Aj,ipj,Tj), j = 1,2, are two independent instances. 

However, to construct an adequate set for /i[V(8)]/ 2 or Ix[VttJ]/ 2 , we need more than 
adequate sets for I\ and / 2 - A set T C P(A) is an extended test set for / = (A,^,!^) 
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if T is adequate for (A, true, T) and T n D is adequate for /. The following lemma and its 
corollary tie together extended and adequate sets. We leave the easy proof to the reader. 

Lemma 5.3 If T and T' are two extended test sets for I then so is (T n D) U (T" \ D). 



Corollary 5.4 If T is a minimal extended test set for I then T C\D is a minimal adequate 
set for I. 

The advantage of working with extended test sets is that extended test sets for Ii[V®]l2 
and Ji[VI±J]i2 can be constructed from extended test sets for I\ and I2, and the construction 
preserves the property of being minimal. For [V<8>] we simply define Ti[V®]T2 = T\ x 
Ti- The definition of Ti[VW]T2 resembles that of Ti[V(8>]72, but it is technically a bit 
more complicated; we omit the details here. With these definitions, Theorem [5.2| holds for 
extended test sets in place of adequate sets and for a = V® or a = Vtfcl. Thus we can extend 
the approach in section |5.3| to large instances formed using [V<S>] and [Vtfcl]. However, the 
operations [V(g>] and [Vttl] do not seem as useful in forming combined instances; typically 
one wishes to use the conjunction, not disjunction, of constraints. 



6 Implementation issues 

We have built a prototype implementation of a tool for generating adequate test sets. The 
tool reads an instance / of the test selection problem, and produces a minimal adequate 
set for /. The instances accepted by the tool are specified in the concrete language of 



Section |3.3| the criteria EACH and EXHAUSTIVE are also allowed, and are automatically 
converted to expressions that use only tfc) and <8>. 
Internally, the tool works in six phases: 

1. Parse the input and check its consistency (only declared parameters and values are 
used, no parameter is declared twice, etc.). 

2. Eliminate EACH and EXHAUSTIVE. 

3. Transform the criterion to the l±l(g) form. 
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declaration 

Alice : { Al, a2, A3, a4, a5 } 

Bob : { Bl, b2, b3, b4, b5 } 

Cathy : { Cl, C2, C3, C4, C5 } 

Diana : { Dl, d2, d3, d4, d5 } 

Elaine : { El, e2, e3, e4, e5 } 

criterion 

EXHAUSTIVE( Alice, Bob, Cathy, Diana, Elaine ) 



Figure 3: An instance to generate 3125 test points 

4. Transform the constraint to the VA form. 

5. Find a minimal adequate set, using the algorithm in Figure |2[ 

6. Print the test points. 

The tool is implemented in C; the total size of the source files is about 1200 lines. The 
basic data structures are trees and forests, which are used to represent the parsed declara- 
tions, constraints and criteria, as well as the intermediate results for the transformations in 
phases || and ||. 

We have tested the tool on RISC System/6000 Model 560, under the AIX operating 
system. [] To measure the execution time on instances with large minimal adequate test 
sets, we have used the criterion EXHAUSTIVE. For the instance in Figure [3|, the domain 
D(ip) has 3125 points, and the only adequate set is the whole domain. Although this is a 
very special form of a test selection criterion, the tool does not take any shortcuts; instances 
like this one are therefore suitable for performance measurements. The execution time of 
the tool for this input is slightly less than 30 seconds — that is, more than 100 test points 
per second. By using more sophisticated data structures we would be able to improve this 
number substantially; however, enhancing the functionality of the tool is more important 
than optimizing its running time. 

^ISC System/6000 and AIX are trademarks of International Business Machines Corporation. 
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In particular, it would be worthwhile to extend the language with other data types (see 
the discussion of future work in Section 7.3). Other possible enhancements would be to add 
heuristics to the test selection algorithm, and to compute bounds on the size of the test set 
before the selection algorithm is invoked. 

Although the tool always produces a minimal adequate set, it makes no attempt to 
come close to a minimum adequate set. A more sophisticated implementation would include 
heuristics to make the generated set smaller in "typical cases". A simple heuristic of this 
kind is to order the subcubes in the set S in Figure § so that smaller subcubes are processed 
before larger ones. 

An approximate bound for the size of the produced test set would be useful as an early 
feedback to the user when the tool is used on a large instance. The user would appreciate 
some estimate of the size of the test set before the test selection algorithm itself is run. An 
upper bound can be easily computed as follows, even before phase ||| begins: In the criterion 
expression, replace each primitive criterion by the value 1, replace each t±J by the operator 
+, and each CED by the operator x. Then evaluate the resulting arithmetic expression; the 
result is an upper bound for the size of the minimal test set produced by the tool. An 
enhanced version of the tool would first display an initial (pessimistic) upper bound on the 
size of the test set, and then update the bound as the computation progresses. The designer 
could abandon execution if the bound seemed hopelessly large. 



7 Concluding remarks 
7.1 Related work 

As is pointed out in the introduction, the representation of test selection criteria by sets 
of subsets of the input domain was considered, implicitly or explicitly, by a number of 
researchers. In partition testing §, the input domain is partitioned into subsets, and one 
test point is then selected in each subset. This is an elaboration of the condition table method 
of Goodenough and Gerhart ||. In this line of research, the emphasis has been on rules 
for constructing criteria from program texts and specifications. In contrast, the emphasis 
in the present paper is on a language for specifying criteria (i.e. sets of subdomains), and 
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on operations that allow test designers to combine criteria. 

In his discussion of functional testing, Howden || stresses the need to identify input 
domains, and gives guidelines for systematic selection of test points for several types of 
input values that occur in scientific programs. Our basic philosophy is similar to Howden's; 
we develop this point of view further, by automating part of the selection process. 

An important technical point is that we do not attempt to represent a criterion by a set 
of disjoint subsets. Note that our operation t±J would make little sense if we only considered 
sets of disjoint subsets. As is explained by Jeng and Weyuker |J, many naturally arising 
test selection criteria lead to non-disjoint sets of subdomains. 

Gourlay || presents a precise framework for the discussion of issues in testing. In his 
terminology, our test selection criteria are a special form of the test methods for the set- 
choice construction testing system. Gourlay reinterprets previously published discussions 
about the suitability of various test selection criteria. In our approach, we do not attempt 
to decide a priori which criteria are sufficient — we leave that decision to the test designer. 
That is why we emphasize the importance of a language in which criteria are specified. 

7.2 Comparison with TSL 

Balcer, Hasling and Ostrand |2| describe a complete test language, called TSL, in which 
the test designer specifies a template for the test cases to be generated, categories (i.e. 
parameters and environment conditions), choices of values for the categories, and results of 
the test cases. A TSL specification is automatically translated to a set of individual test 
cases. 

We now explain how TSL relates to the languages for test selection criteria that we 
propose in this paper. We will not describe TSL here; the reader is referred to the original 
paper for a detailed description. 

A TSL specification contains declarations of parameters, each with a set of values. (TSL 
makes a distinction between parameters and environment conditions, but for the purpose of 
this discussion both are considered to be parameters.) The specification also contains a set 
of Boolean conditions (the IF clauses in the RESULT sections), which are used to decide 
what combinations of parameter values are to be selected to form test cases. There are two 
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types of such conditions: unqualified ones, and those qualified by the directive SINGLE. 

Let us first consider the following simplified form of the test selection criterion used 
by TSL: For an unqualified condition, all combinations of parameter values satisfying the 
condition should be selected. For a qualified condition, at least one combination of param- 
eter values should be selected. We show how to specify this criterion in our language. Let 
cpi,..., (p m be the unqualified conditions, and let ai,...,a r be the conditions qualified as 
SINGLE. The test selection criterion is 



l+j ( (&) ® EXHAUSTIVE ) a [+j (a,) (11) 
i=l i=i 



If all ipi and Gj are conjunctions of conditions of the form 

q = c 



where q is a parameter and c is a value of q, then the criterion (11) can be expressed in the 



concrete language from Section 3.3. 



The TSL criterion as stated in || is actually more complicated than the one in the 
previous paragraph. An error-sensitizing rule is used to constrain the choice of a test point 
for (<Tj). The rule is described only informally in Q; we now state one possible formalization, 
using our language. For each dj, j = 1, . . . , r, let Uj be the disjunction of all (pi and <Tj in 
the same RESULT section, other than Oj itself. The modified test selection criterion is 



m r 

l+j ( (cpi) (g) EXHAUSTIVE ) tt) 1+J (ajA^ujj) 

i=l 3=1 

It is not our goal to discuss the merits of various versions of the error-sensitizing rule. We 
merely make the point that our language is a convenient notation for stating such rules 
precisely. 

The language scheme proposed in this paper indicates the direction in which the TSL 
notation for test selection, and other similar notations, could be extended. The test designer 
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would benefit from the flexibility of the operations ttl and ®. For instance, in the example in 
Section |2], suppose that the test designer wants to fix separator _1 = "/", separator _2 = 
"/" and string_l_occurs = true, and test all values of string_l except "" and all values 
of string_2 at least once, but not necessarily all combinations of string_l and string_2. 
The criterion to express that requirement is 

(separator_l = "/"} ® (separator_2 = "/") <8> (string_l_occurs = true) 
® (string_l / ""} ® ( EXHAUSTIVE(string_l) bfcl EXHAUSTIVE(string_2) ) 

7.3 Future work 

Here we mention several topics for further research which we have not addressed in the 
present paper. We group the topics into two categories: Improved algorithms for the 
concrete language, and extensions of the language and its use. 
[> Algorithms for our concrete language 



In Section 5.3 we describe an algorithm for discovering a decomposition into independent 
instances. We assume that the constraint has the form AfcV'fc- To what extent can that 
assumption be relaxed? 

Consider only the instances of the test selection problem that are built from instances of 
some small bounded size using the operations [A®] and [A ttl]. Is there an efficient algorithm 
for finding minimum adequate sets for the instances in this special form? 

Heuristics for finding "almost-minimum" adequate test sets for "common" test selection 



criteria should be investigated. In view of the results in Section 4.2, known heuristics for 
graph coloring would be a good starting point. 
> Extensions of the language 



The general language schema in Section 3.1 is a framework for further design of concrete 



languages based on other data types. After the enumerated data types treated in Section 3.3, 
the next most important type is integers. Some useful criteria for integers were mentioned 
in |pH ], but we have not studied in detail the algorithms needed to deal with those criteria. 

Another important candidate for incorporation into the general schema is the type 
words over a finite alphabet, which would be useful for specifying criteria that have to 
do with control flow in a program or in a state machine. 
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The ideas in Section 5.3 lead naturally to modular descriptions of complex test suites. In 
a testing system supporting modularity, parameterized test cases along with test selection 
criteria could be created for various subsystems of a complex implementation under test, 
independently of each other (perhaps written by different test designers), and then combined 
by means of simple operators. 
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